A cluster is a group of devices that are connected together in a network. A typical cluster includes one or more devices that perform data processing operations (e.g., a mainframe), and one or more devices that do not perform data processing operations (e.g., a disk). The devices that perform data processing operations are considered processing devices or nodes, and the devices that do not perform data processing operations are considered non-processing devices.
It is common for the processing devices of a cluster to share one or more resources. When the processing devices share a resource, each processing device cooperates with the other processing devices to coordinate use of the shared resource. For example, the processing devices of a particular cluster may share a storage disk. The processing devices may cooperate with each other by coordinating access to storage locations on the storage disk.
Devices prefer to join clusters rather than form clusters. That is, when a processing device boots, the processing device attempts first to join a cluster if one exists, and if one does not exist, the processing device attempts alternatively to form a new cluster. Devices typically form a cluster by contributing votes to a cluster vote count. When the devices contribute enough votes so that the vote count reaches a threshold amount called a quorum, the devices form a cluster. Devices that contribute one or more votes are considered voting devices. Devices that contribute zero votes or devices that do not vote are considered non-voting devices. In general, a voting device of a cluster may be a processing device (e.g., a voting node) or a non-processing device (e.g., a quorum disk).
Quorum will now be discussed in further detail. Cluster administrators typically set the quorum of a cluster equal to more than one half of the total number of votes available to the cluster. That is, the cluster administrator generates a total number of votes available to the cluster by counting the votes provided by all of the devices that are configured to form or join the cluster. Then, the cluster administrator sets the quorum equal to a value that is greater than one half the generated total number.
The processing devices typically do not perform data processing operations until the cluster is formed, i.e., until the cluster vote count reaches quorum. Prior to formation of the cluster, the processing devices generally have limited operability such as the capability to boot, to contribute votes to the cluster vote count, and to transfer device information from one device to another. Once the cluster is formed, the processing devices perform data processing operations, and devices may join or leave the formed cluster. When a device joins the cluster, any votes of the joining device are added to the cluster vote count. Similarly, when a device leaves the formed cluster, any votes of the leaving device are removed from the cluster vote count. That is, the cluster recalculates the vote count when a device joins or leaves so that the vote count includes only the votes of the devices presently in the cluster.
When one or more devices leave the cluster and the cluster vote count is recalculated, it is possible for quorum to be lost. That is, the recalculated vote count may drop below quorum. When a cluster loses quorum, the devices remaining in the cluster enter a "hanging" state, and cease performing data processing operations. The devices do not resume data processing operations unless one or more devices join the cluster and contribute enough votes so that the vote count reaches quorum again. When quorum is restored to the cluster, the hanging processing devices complete any data processing operations that were left uncompleted.
Cluster partitioning will now be explained. A cluster may include two sets of devices that become disconnected from each other (e.g., due to a failure somewhere in the network). When disconnection occurs, each set becomes a separate cluster such that a first new cluster and a second new cluster exist simultaneously. That is, from the perspective of the first set of devices (the first new cluster), the devices of the second set have left the original cluster. Similarly, from the perspective of the second set (the second new cluster), the devices of the first set have left. Such separation of the original cluster into multiple new clusters is called cluster partitioning.
It should be understood that, in general, two partitioned clusters cannot perform data processing operations simultaneously. In particular, since quorum equals more than one half of the total votes available to the original cluster, only one of the two new clusters can have quorum at any given time. If the first set of devices has enough votes to maintain quorum, the second set will not have enough votes for quorum. In this situation, the first set continues to perform data processing operations while the second set hangs. Alternatively, if the second set of devices has enough votes to maintain quorum, the second set continues to perform data processing operations while the first set hangs. After partitioning, it is possible that neither of the two new clusters has enough votes to maintain quorum such that both new clusters hang.
Some clusters use generation numbers to track cluster operation. In such a cluster, the processing voting devices typically store a current cluster generation number in dedicated non-volatile memory locations (e.g, local non-volatile memory). When a processing voting device leaves the cluster, the remaining processing voting devices typically increment the cluster generation number, and store the incremented cluster generation number in place of the original cluster generation number. The processing voting device that leaves typically does not increment its cluster generation number and continues to store the original cluster generation number. Accordingly, the processing voting devices that remain in the cluster should always have a cluster generation number that is greater than or equal to that of a processing voting device that has left the cluster. If a cluster's current cluster generation number is less than that of a processing voting device that has not joined the cluster, the cluster has become operational inadvertently, and the cluster should be stopped from operating as quickly as possible (e.g., crashed) to avoid destroying cluster information.
When a processing voting device joins an already formed cluster, the processing voting device typically compares its stored generation number with the current cluster generation number (provided by a processing voting device of the cluster). If the stored generation number of the processing voting device is less than or equal to the current cluster generation number, the processing voting device typically joins the cluster and stores the current cluster generation number in place of its stored generation number. However, if the stored generation number of the processing voting device is greater than the current cluster generation number, the processing voting device sends a signal to the cluster indicating that its generation number is higher than the current generation number of the cluster. The processing devices of the cluster generally crash when the cluster receives the signal so that any damaging data processing operations performed by the processing devices of the cluster are discontinued.
Examples of clusters include OpenVMS clusters and UNIX TruClusters made by Digital Equipment Corporation of Maynard, Massachusetts, clusters made by Sequent Computer Systems, Inc. of Beaverton, Oregon, and clusters made by Tandem Computers of Austin, Texas.
Some conventional clusters are susceptible to cluster partitioning into multiple new clusters. Subsequent operation of such new clusters at different times can destroy cluster information on a shared resource in some situations.